1. Field of the Invention
The present invention relates to image processing method and apparatus which correct image distortions caused by an oblique image pickup in which an original image on the object plane is taken from an oblique direction. Further, the present invention relates to image processing method and apparatus which create a composite image from a plurality of partially overlapping images such that the image distortions caused by the oblique image pickup are corrected.
The image processing method and apparatus of the present invention are applied to image capturing and image composition techniques using digital still cameras or digital video cameras, non-contact handy scanners, and other digital imaging facilities.
2. Description of the Related Art
With a rapid progress in the recent development of computer networks, there is an increasing demand for quickly acquiring important information in all aspects of computer systems. Attention is being given to portable image input devices that can easily and accurately capture an image of a document or a photograph and transfer it into the storage of a computer system. As the practical application, an image input system, such as a digital still camera, is used to capture a document image with a size of A4 or the like by taking the original image on the object plane (or the paper).
However, when the original image on the object plane is taken by the image input system from an oblique direction with the image pickup plane of the image input system being not parallel to the object plane, image distortions may take place due to the oblique image pickup. The readability of the captured information will be deteriorated, and it is desired to correct such image distortions. Hereinafter, the image distortions of this kind will be called the oblique-type image distortions.
Several image processing techniques for correcting the oblique-type image distortions are known. For example, Japanese Laid-Open Patent Application No. 3-94383 discloses an image processing apparatus that is adapted for correcting image distortions. In the image processing apparatus, a fixed shape pattern is assumed. An input image that may be captured with an oblique image pickup or the like is arranged within the fixed shape pattern, and the distortions of the input image with respect to the fixed shape pattern are detected. The transformation parameters for restoring the input image into the original shape on the object plane are calculated in accordance with the detected distortions. The image distortions are corrected by using the calculated transformation parameters.
However, the image processing apparatus, disclosed in the above publication, requires the arrangement of the input image within the fixed shape pattern. The operations to capture an image on the object plane and arrange it within the fixed shape pattern are complicated, and it is difficult to provide easy image pickup operations to take an image on the object plane using an image input system.
Japanese Laid-Open Patent Application No. 5-101221 discloses an image distortion correcting device for correcting image distortions. In the image distortion correcting device, an orthogonal grating pattern is applied onto the object plane, and an image on the object plane is captured by using an image input system. Three-dimensional (3D) coordinates of the individual lattice points within the captured image are calculated, and they are transformed into 3D coordinates in the coordinate system fixed to the image pickup plane of the image input system, through an orthogonal projection. Further, the resulting coordinates on the image pickup plane are transformed into two-dimensional coordinates on a two-dimensional plane through a perspective projection. The resulting coordinates are approximated to integer coordinates, which correspond to the picture elements of an output image such that the image distortions in the output image are corrected.
However, the image distortion correcting device, disclosed in the above publication, requires the overlapping of the orthogonal grating pattern onto the object plane and the manual inputting of the two-dimensional coordinates of the respective lattice points in the grating pattern before the calculation of the 3D coordinates of the lattice points. The operations to overlap the grating pattern onto the object plane and manually input the 2D coordinates of the lattice points are complicated and time consuming, and it is difficult to provide easy image pickup operations to take an image on the object plane using an image input system.
Japanese Laid-Open Patent Application No. 9-289600 discloses a digital camera provided with a tilt angle setting unit and a distance measuring unit. The tilt angle setting unit inputs a tilt angle of the CCD image pickup plane to the plane of the object being photographed. The distance measuring unit inputs a measured distance between the digital camera and the object. An image of the object is taken from the oblique direction according to the title angle by using the digital camera. Then, the captured image is corrected into a pseudo front-view image through calculations based on the input tilt angle and the input distance.
However, in the digital camera, disclosed in the above publication, it is very difficult for the operator of the digital camera to manually input an accurate tilt angle between the image pickup plane and the object plane by using the tilt angle setting unit before taking the image of the object. As a result, when the digital camera disclosed in the above publication is used, it is difficult to correct, with high accuracy, the oblique-type image distortions in the captured image.
Japanese Laid-Open Patent Application No. 11-98485 discloses an image input system including a camera which is adapted to reproduce the geometrical configuration of the object based on camera direction information detected from the camera directing angle in the 3D coordinate system camera fixed to the rotation axis, so as to correct the oblique-type image distortions. The distance from the camera to the plane of the object, and the tilt angle of the camera's image pickup plane to the object plane are determined depending on the mechanical structure and layout of the elements of the camera.
The image input system, disclosed in the above publication, requires that the camera be fixed at a constant height with a camera fixing mechanism and be configured such that the camera is rotatable about the rotation axis. The cost of the entire image input system will be increased by such requirements. Further, there is a problem in that the user's operational conditions for taking an image on the object plane are significantly restricted by the mechanical structure of the camera fixing mechanism. It is difficult that the image input system of the above publication be applied to portable image input systems with ease of operation and flexibility.
Further, it is demanded that images of large-size objects, such as a newspaper, having a size larger than A4 size, be captured with a high resolution by using a portable image input device. For this purpose, an image processing technique is proposed in which a composite image is created with a certain high resolution by combining together a plurality of partially overlapping images which are taken by using an image input device. In cases in which the object is assumed to be planar, the image processing technique applies geometrical transformation formulas, such as affine transformation or perspective affinity, for the combination of a plurality of partially overlapping images so as to create a composite image. The technique is shown in “Computer Vision—Technological Comments and Future Trends” by T. Matsuyama et al., published in June 1998 by New Technology Communications Co., Ltd. in Japan.
The basic concept of the above-described technique is that the respective split images are transformed into partial object images on a standard image surface, and they are combined together to form a composite image. However, there is a problem that if the partial object images on the standard image surface contain the oblique-type image distortions, the resulting composite image also contains the image distortions.
Specifically, FIG. 37 shows the creation of a composite image according to the above-described technique. As shown in FIG. 37, a split image “I” is taken from a part of the object on the object plane by using an image pickup device, and the split image “I” is mapped into a partial object image on an extended portion of a standard image surface “Io”. Such partial object images on the standard image surface “Io” are combined so as to form a composite image. The resulting composite image is equivalent to an image that is directly formed by taking the same object by using a wide-angle optical system, the wide-angle optical system allowing the standard image surface “Io” to be extended to cover the overall range of the object. The composite image created by the above technique indicates more perspective details than the image created by a normal shot, and the composite image created by the above technique is more likely to contain the oblique-type image distortions.